Disputation
16 April 2024
University of Mannheim
“[…] introducing various methods of classifying data from open-ended survey questions and empirically illustrating their application.
A central research question addressed in this thesis therefore concerns the analysis of (short) text data generated by open-ended survey questions.” (Landesvatter 2023, p.2)
“survey questions that do not include a set of response options” (Züll, 2016, p. 1)
“require respondents to formulate a response in their own words and to express it verbally or in writing” (Züll, 2016, p. 1)
≠ closed-ended questions with answer categories presented in a closed form (Inui et al., 2001, p. 1)
“[…] introducing various methods of classifying data from open-ended survey questions and empirically illustrating their application. A central research question addressed in this thesis therefore concerns the analysis of (short) text data generated by open-ended survey questions.”
Table 1. Overview of methods for classifying open-ended survey responses
➡️ The increase in methods to collect open-ended answers (e.g. smartphone-administered surveys, voice technologies, novel methods) calls for testing and validating automated methods to analyze the resulting data
❓ Why did I chose the Survey Context?
❓ Why do I focus on computational methods?
Figure 1: The previous question was: ‘How often can you trust the federal government in Washington to do what is right?’. Your answer was: ‘[Always; Most of the time; About half of the time; Some of the time; Never; Don’t Know]’. In your own words, please explain why you selected this answer.
fully manual methods require high ressources (time and effort)
but more importantly, human codings can
automated methods offer objectivity and systematicness (Zhang et al., 2022)
still, issues persist (e.g., transparency) which makes it crucial to test and evaluate methods for the social sciences
| 1 | 2 | 3 |
|---|---|---|
| How valid are trust survey measures? New insights from open-ended probing data and supervised machine learning | Open-ended survey questions: A comparison of information content in text and audio response format | Asking Why: Is there an Affective Component of Political Trust Ratings in Surveys? |
Co-authored by: Dr. Paul C. Bauer
Published In: Landesvatter, C., & Bauer, P. C. (2024). How Valid Are Trust Survey Measures? New Insights From Open-Ended Probing Data and Supervised Machine Learning. Sociological Methods & Research, 0(0). https://doi.org/10.1177/00491241241234871
Co-authored by: Dr. Paul C. Bauer
Submitted to: Public Opinion Quarterly in February 2024
Co-authored by: Dr. Paul C. Bauer
Submitted to: American Political Science Review in March 2024
Operationalization via sentiment and emotion analysis
Transcript-based
Speech-based
web surveys can be used to collect narrative answers that provide valuable insights into survey responses
various modern developments (smartphone surveys, speech-to-text algorithms) can be leveraged to collect such data in innovative ways (e.g., spoken answers)
computational measures can be applied to classify open-ended answers from surveys in order to inform ongoing debates in different fields, e.g.:
Facilitated accessibility and implementation of semi-automated methods.
supervised models have been a standard in automated methods, but recent developments of large and general-aim pre-trained models (e.g., BERT) allow less resource-intensive fine-tuning
For example, using only ~13% (1,000 documents from 7,500 in Study 1) documents for fine-tuning resulted in sufficient accuracy (i.e., 87%)
Increase in possibilities of fully automated methods (e.g., prompt engineering.
Landesvatter: Methods for the Classification of Data from Open-Ended Questions in Surveys